1. Field of the Invention
This invention is related to address spaces in multi-node systems and, more particularly, to mapping local and global addresses.
2. Description of the Related Art
In many systems, multiple nodes are coupled together to form the system. Each node may comprise an integrated circuit, or multiple integrated circuits and/or other devices (e.g. input/output (I/O) devices and/or interfaces). Each node has a local address space used to address memory in the node or coupled to the node (“local memory”), as well as various I/O devices or interfaces in the node. Typically, the address spaces are relatively fixed (e.g. various regions within the address space are dedicated to local memory or I/O devices). There may be some amount of programmability to the address space (e.g. regions of the address space mapped to memory may be sized to permit varying amounts of local memory).
The address space within each node of a multi-node system typically matches. That is, given the same amount of memory and the same I/O devices, the same numerical addresses in each local address space refer to the local memory or I/O devices in that node. Accordingly, sharing local memory or I/O devices with other nodes (permitting the other nodes to access and/or update the shared local memory or devices) is complicated. One typical mechanism is shown in FIG. 1.
In FIG. 1, a local address space 10 corresponding to a first node (node 0), a local address space 12 corresponding to a second node (node 1), and an I/O address space 14 corresponding to an I/O interface used to communicate between node 0 and node 1 are shown. Address 0 is at the bottom of each address space in FIG. 1. Each local address space has a variety of regions (e.g. a local I/O region for the local I/O devices and interfaces in each node, a memory region for the local memory, and an external region which is mapped onto the I/O address space 14). While contiguous regions are shown in FIG. 1 for simplicity, multiple local I/O regions and/or memory regions may be defined in each local address space 10 and 12.
A shared memory location 16 in the node 1 local address space is also illustrated, via the cross-hatched box in the node 1 local address space 12. A shared I/O location (e.g. corresponding to a local I/O device or interface that is to be shared between the nodes) may be similar. The shared memory location 16 is addressed using an address A in the node 1 local address space 12. The address A cannot be used by node 0 to access the shared memory location 16, as the address A is in the memory region of the local address space 10 and refers to a local memory location 18 in the node 0. For node 0 to access the shared memory location 16, an address in the external region must be used (to cause a transaction on the I/O interface to communicate to node 1). Thus, for example, an address B in the external region at the local address space 10 may be assigned to the shared memory location 16. The address B is further mapped to an address C in the I/O address space 14, which is mapped to the address A in the local address space 12 by the node 1 in response to receiving the I/O transaction on the I/O interface.
In the illustrated mechanism, three different addresses (A, B, and C) are used to access the same memory location 16. If additional nodes (not shown) access the same memory location, even more addresses may be used. Such a scheme may create complexities for software executing on the system. For example, if a software process that accesses the memory location 16, and the process migrates from one node to another, the address used to access the memory location 16 must be recalculated. To perform the recalculation properly, the process must be “aware” of which node it is running on, which may complicate the process. Some currently existing software assumes that a given local address in the external region of the local address space is numerically equal to the corresponding I/O address in the I/O address space (although it clearly cannot be equal to the address in the other local address space, if a shared memory location or I/O device is being accessed in another node). Such assumptions further complicate address space management. In nodes in which virtual address spaces are implemented (e.g. nodes having processors), some software may even attempt to make the virtual address, the corresponding physical address in the local address space, and the corresponding I/O address numerically equal.
The illustrated mechanism also presents difficulties if cache coherency is to be maintained for the shared memory location. Typically, coherency schemes rely on comparing the addresses of transactions to the cached addresses in a given cache. However, if each node is using different addresses to access the same location, comparing the addresses is insufficient to detect an access to the same memory location as a cached memory location. Some multi-node cache coherent nonuniform memory access (CC-NUMA) systems use the most significant address bits as a node identifier identifying the node to which the address is mapped. Such systems typically design the interconnect between nodes to support a global address space that is shared by the nodes (e.g. the “local” address spaces are merely part of the global address space that is assigned to the node).